This invention relates to an image searching method and an image processing method. More specifically, this invention relates to an image searching method in which specific information extracted from an image such as simple figures or subjects such as human individuals or physical objects or specific information that accompanies the image such as audio information or message information such as characters and marks is stored in a database as accessory information for the image and images are searched through on the basis of the stored accessory information.
The present invention also relates to an image searching method which extracts the subjects of interest on the basis of the position information, camera information and map information that are relevant to the time when the image was taken and which uses the result of extraction (in the form of image) to search through images.
The invention also relates to an image processing method which utilizes such image searching methods or the specific information extracted from the image.
When viewing a portrait, we pay the most attention to the face of the person in the picture. Therefore, when the original image typically recorded on a film is to be printed on a duplicating material such as print paper, the amount of exposure has to be determined such that the face of the person is reproduced in an appropriate color and density.
To meet this requirement of portraiture, Unexamined Published Japanese Patent Application (kokai) No. 346332/1992 proposed a method comprising the steps of dividing an original color image into a multiple of pixels, separating each pixel into three colors R, G and B, performing photometry, constructing histograms about hue and saturation on the basis of the result of photometry, slicing as a human face equivalent area those regions of the histograms which contain pixels having a hue and saturation that correspond to the face of the person in the portrait, and determining the correct exposure on the basis of the photometric data for the sliced regions.
In order to increase the probability for extracting the human face equivalent area, Unexamined Published Japanese Patent Application (kokai) No. 160993/1994 proposed various techniques such as removing the area contiguous to the edges of the image as the background region or transforming the extracted area into a figure defined by lines and determining whether the extracted area corresponds to the face of the person in the portrait on the basis of the shapes of neighboring areas surrounding the extracted area and the shape of the extracted area.
These techniques, however, have a common problem in that if the original image contains the ground, trees and any other areas having a hue and saturation close to those of flesh color, such areas may be erroneously interpreted as the human face equivalent area. If these areas are adjacent the true human face equivalent area, they cannot be correctly separated from the latter, with the result that the original image is not likely to be divided into areas of appropriate sizes.
If the areas other than the face of the person in the portrait are erroneously interpreted as the human face equivalent area or if the areas adjacent the human face equivalent area which have a similar hue and saturation cannot be separated from the latter, it has been impossible to determine the correct exposure for appropriate printing of the true human face.
To solve this problem, it would be effective to utilize not only the color (hue and saturation) of the area of interest but also the information about its shape. Speaking of the human face equivalent area, it will in most cases have an elliptical or oval shape, so this special shape may be added to the color information as a condition for extracting the human face equivalent area.
Thus, the information about the shapes of subjects in the original image is very important to the purpose of image detection or searching. In this connection, reference may be had to the technology disclosed in commonly assigned Unexamined Published Japanese Patent Application (kokai) No. 138471/1997. In this technology, the edges of an image are detected and the lines of contours of an area corresponding to the person in portrait are extracted so that the area, particularly the one corresponding to the face of the person, is rendered to reproduce the preferred color.
According to another prior art image searching system, the image the searcher wants to extract can be searched for without requiring the searcher to draw the image of interest or even if no image close to the one wanted by the searcher is ready at hand or within the list of available images. For this image searching system, reference may be had to Unexamined Published Japanese Patent Application (kokai) Nos. 21198/1995 and 249352/1996.
For use in taking documentary pictures by laymen, cameras have recently been put on the commercial market that provide automatic recording of xe2x80x9cwhen and wherexe2x80x9d the picture was taken. Those cameras are a composite product in which an artificial satellite based global positioning system (GPS) commonly incorporated in car navigation systems is combined with a direction sensor and a camera [Shashin Kogyo (Photographic Industry), 1996-7, pp. 12-15, 84-87).
While photography has several roles to play, the importance of photo""s documentary and evidential performance is one of the essential values of photograph. This is why accessory information telling xe2x80x9cwhen and wherexe2x80x9d the subject was shot is important to documentary pictures. Conventional cameras have so far provided the information about xe2x80x9cwhenxe2x80x9d to some extent but not the information about xe2x80x9cwhenxe2x80x9d.
In this sense, the camera described as a new product in Shashin Kogyo, supra (which is hereunder sometimes referred to as a GPS incorporating camera) may well be considered a device that can fulfil the inherent functions of documentary pictures. First of all, the dating function of the conventional xe2x80x9cdatingxe2x80x9d camera and the representation of time by the GPS incorporating camera are entirely different in terms of evidential value: the former provides time that can be determined xe2x80x9carbitrarilyxe2x80x9d (for personal use) whereas the latter records the xe2x80x9cofficial timexe2x80x9d being sent from the satellite.
Exactly speaking, the position information provided by the GPS incorporating camera does not tell the position of the subject but indicates the position of the camera and the direction of shooting. Therefore, if the picture is a long shot, the position of the subject is not always definite. Even in this case and if there is a need, the position of the subject can be correctly calculated from, for example, the distance between that subject and another subject in the image.
The present inventors previously proposed in Japanese Patent Application No. 162130/1998 a photographic printing method comprising the steps of acquiring at least one kind of shooting information in association with the image taken that is selected from among shooting position, shooting direction and magnification rate, obtaining additional information associated with at least a portion of the acquired shooting information and adding the obtained additional information to a photographic print. The present inventors also proposed a photographic printing apparatus implementing the method.
Using the photographic printing method and apparatus, the customer looking at the picture can be easily reminded of relevant atmosphere such as the circumstances of shooting on the basis of the shooting information added in association with the recorded picture. Thus, the method and apparatus can not only enhance the documentary function of photographic prints but also allow them to serve the purpose of amusement (by furnishing a topic of conversation).
The present invention is a further development of the concept of the above-described prior art for detecting areas of interest using specific shapes in the original image as a key. It is therefore the first object of the invention to provide an image searching method capable of efficient image detection or search (which is hereunder collectively referred to as image search) using as a key the shapes of subjects such as a person or a physical object in the original image or specific information that accompanies the original image such as audio information or message information.
The present invention has been accomplished with a view to further facilitating the use of this technology. It is therefore the second object of the invention to provide an image searching method that is applicable when the customer wants to locate a desired image from a number of stored images and which enables easy image search by utilizing the position information, camera information and map information that are relevant to the time when the image was taken and which is added in association with the recorded image.
The third object of the invention is to provide an image processing method which utilizes such image searching methods or the specific information extracted from the original image.
In order to attain the above overall object, the present invention provides an image searching method, comprising the steps of:
extracting or recognizing specific information for an image that either exists in the image or accompanies it or both exists in the image and accompanies it; allowing either the specific information as-obtained or related information relative to the specific information to be appendant to image data for the image; storing the image data as well as either information appendant to the image data as accessory information in a database; designating at least a portion of the accessory information as a searching condition; searching through the database under the designated searching condition; determining a degree of agreement with the accessory information appendant to a selected image data; and reading an image having the degree of agreement not lower than a predetermined value
In order to attain the first object, the present invention provides an image searching method, comprising the steps of: applying to an image a treatment of extracting M (Mxe2x89xa71) kinds of specific geometric figures predetermined for the image; allowing to be appendant to image data of the corresponding image an accessory information containing a combination of position and size or a combination of position, size and direction or a combination of position, size and color or a combination of position, size, direction and color that correspond to N (Nxe2x89xa70) specific geometric figures that have been obtained as a result of extraction; storing the image data as well as the accessory information appendant to the image data in a database; designating, for L (L is from one to M) kinds of the specific geometric figures, presence or absence of the specific geometric figures or an outline of at least one element selected from among the position, size, direction and color; determining a degree of agreement with the accessory information appendant to image data of the stored image; and reading an image having the degree of agreement not lower than a predetermined value.
Preferably, the specific information existing in the image is M (Mxe2x89xa71) kinds of specific geometric figures predetermined for the image, the related information to be stored as the accessory information is either a combination of position and size or a combination of position, size and direction or a combination of position, size and color or a combination of position, size, direction and color that correspond to N (Nxe2x89xa70) specific geometric figures that have been obtained as a result of extraction by processing the image with a treatment of extracting the M kinds of the specific geometric figures, and the designating step of at least a portion of the accessory information as the searching condition comprises designating, for L (L is from one to M) kinds of the specific geometric figures, presence or absence of the specific geometric figures or an outline of at least one element selected from among the position, size, direction and color.
Preferably, the specific information existing in the image is M (Mxe2x89xa71) kinds of specific subjects predetermined for the image, the related information to be stored as the accessory information is either a combination of position and size or a combination of position, size and direction or a combination of position, size and color or a combination of position, size, direction and color that correspond to N (Nxe2x89xa70) specific subjects that have been obtained as a result of extraction by processing the image with a treatment of extracting the M kinds of the specific subjects, and the step of designating at least a portion of the accessory information as the searching condition comprises designating, for L (L is from one to M) kinds of the specific subjects, presence or absence of the specific subjects or an outline of at least one element selected from among the position, size, direction and color.
Preferably, the image is temporally continuous images, the specific information existing in the image is a specific geometric figure predetermined for the image, the related information to be stored as the accessory information comprises information about the specific geometric figure and a direction of movement of an image corresponding to the specific geometric figure that have been obtained by processing the temporally continuous images by a treatment of extracting the specific geometric figure from the temporally continuous images at predetermined intervals of time or frame numbers and determining the direction of movement of the image corresponding to the geometric figure based on comparison of a position of the extracted specific geometric figure between frames, and the step of designating at least a portion of the accessory information as the searching condition comprises designating, for the temporally continuous images, outlines of the specific geometric figure and the direction of its motion vector at the predetermined intervals of time or frame numbers.
Preferably, the image is temporally continuous images, the specific information existing in the image is a specific subject predetermined for the image, the related information to be stored as the accessory information comprises information about the specific subject and a direction of movement of an image corresponding to the specific subject that have been obtained by processing the temporally continuous images by a treatment of extracting the specific subject from the temporally continuous images at predetermined intervals of time or frame numbers and determining the direction of movement of the image corresponding to the specific subject based on comparison of a position of the extracted subject between frames, and the step of designating at least a portion of the accessory information as the searching condition comprises designating, for the temporally continuous images, outlines of the specific subject and the direction of its motion vector at the predetermined intervals of time or frame numbers.
Preferably, the image has audio information, the specific information accompanying the image is the audio information, the accessory information is the audio information that has been recognized for the image having the audio information, the step of designating at least a portion of the accessory information as the searching condition comprises designating an outline of audio information for the image having the audio information, and the degree of agreement is with speech recognition information that is appendant to the stored image data.
Preferably, the accessory information comprises not only the audio information but also the related information which is either a combination of position and size or a combination of position, size and direction or a combination of position, size and color or a combination of position, size, direction and color which correspond to a specific geometric figure, and the searching condition comprises not only the audio information but also presence or absence of the specific geometric figure or at least one element selected from among position, size, direction and color which correspond to the specific geometric figure.
Preferably, the accessory information comprises not only the audio information but also the related information which is either a combination of position and size or a combination of position, size and direction or a combination of position, size and color or a combination of position, size, direction and color which correspond to a specific subject, and the searching condition comprises not only the audio information but also presence or absence of the specific subject or at least one element selected from among position, size, direction and color which correspond to the specific subject.
Preferably, the specific information which both exists in and accompanies the image is person""s information comprising a facial image of a particular person and his or her name, the related information to be stored as the accessory information is presence or absence of the particular person or either a combination of position and size of the particular person or a combination of position, size and direction of the particular person or a combination of position, size and color of the particular person or a combination of position, size, direction and color of the particular person and obtained by the steps of preliminarily registering person""s information comprising the facial images of the A particular persons (Axe2x89xa71) and their names and performing extraction of persons and a person identifying treatment using the person""s information at the time of storing the image, the step of designating at least a portion of the accessory information as the searching condition comprising designating presence or absence of B particular persons (B is from one to A) or at least one element selected from among positions, sizes and directions of the particular persons, and the degree of agreement is with the accessory information to each of the stored images.
Preferably, the image is temporally continuous images, and the related information to be stored as the accessory information is obtained by processing the temporally continuous images by the person identifying treatment using the person""s information at predetermined intervals of time or frame numbers and comprises information about the particular parsons and a direction of the movement of the particular parsons, the step of designating at least a portion of the accessory information as the searching condition comprising designating presence or absence of the particular persons or at least one element selected from among positions, sizes and colors of the particular persons, and the direction of its motion vector.
Preferably, the image has audio information, the accessory information comprises the related information obtained for the image having the audio information by a person identifying treatment using the person""s information and audio information recognized for the image having the audio information, and the step of designating at least a portion of the accessory information as the searching condition comprises designating an outline of the related information containing the presence or absence of the particular persons or at least one element selected from among positions, sizes, directions and colors or the audio information as the searching condition applicable to the image having the audio information.
Preferably, the image has message information, the specific information accompanying the image is the message information, the accessory information is the message information recognized for the image having the message information, the step of designating at least a portion of the accessory information as the searching condition comprises designating an outline of message information for the image having the message information, and the degree of agreement is with message recognition information appendant to the stored image data.
Preferably, the message information is text data or appendant image data.
Preferably, the image is a computer graphics image, the specific information appendant to the image is editing data used to create the image, the accessory information is the editing data, the step of designating at least a portion of the accessory information as the searching condition has the substeps of designating a scene of the computer graphics image by at least one element selected from among position, size and direction of at least one member selected from among a specific geometric figure, a specific physical object and a specific color and automatically transforming the designated scene designation data to a computer graphics command, and the step of determining the degree of agreement is for determining the degree of agreement between the resulting computer graphics command and the editing data appendant to a selected item of the stored image data.
Preferably, the computer graphics image is temporally continuous computer graphics images, the accessory information comprises not only the editing data but also either the specific geometric figure or the specific physical object or both and a direction of movement of an image that have been obtained by determining, for the temporally continuous computer graphics images, the direction of movement of the image corresponding to either the specific geometric figure or the physical object or both at predetermined intervals of time or frame numbers based on comparison between frames of the position of either the specific geometric figure or the specific physical object or both, and outlines of either the specific geometric figure or the specific physical object or both and the direction of their motion vector are designated as the searching condition at predetermined intervals of time or frame numbers for the temporally continuous computer graphics images.
Preferably, the step of designation has the substeps of designating at least a portion of the accessory information by at least one means selected from among symbols, language and speech and automatically transforming at least one designated element selected from among symbols, language and speech into the specific information or accessory information therefor.
Preferably, an area in the image data that meets the searching condition is displayed in temporal preference.
Preferably, an area in the image data that meets the searching condition is displayed as enhanced by a different image processing scheme than applied to the other areas.
Preferably, the image data is processed in accordance with a display performance of a display terminal.
Preferably, the image areas that have been searched through are managed area by area and a priority of reference in search mode is set in accordance with access frequency.
Preferably, the image areas that have been searched through are managed area by area and an image resolution for data storage is set in accordance with access frequency.
In order to attain the second object, the present invention provides an image searching method, comprising the steps of; extracting a name of a shot place where an image was shot or a subject and their position on a map; storing image data of the image as well as an extracting result allowed to be appendant to the image data as accessory information: designating the position or the name of the shot place or the position or the name of the subject as reference data; determining a degree of agreement with the accessory information appendant to image data of the stored image; and
reading an image having the degree of agreement not lower than a predetermined value.
Preferably, the specific information accompanying the image is a name of a shot place where an image was shot or a subject and their position on a map, the accessory information is a result obtained by extracting the name of the shot place or the subject and theirs position on the map from shooting information when the image was shot and map information, and the step of designating at least a portion of the accessory information as the searching condition comprises designating the position or the name of the shot place or the position or the name of the subject as reference data.
Preferably, the shooting information is acquired in association with the shot image and is either a first data composition consisting of shooting position or a second data composition consisting of shooting position and shooting direction or a third data composition consisting of shooting position and magnification rate or a fourth data composition consisting of shooting position, shooting direction and magnifying power of shooting.
Preferably, an outline of depth is designated as reference data concerning the shooting position and subject""s position by arranging patterned subjects in a virtual map space.
Preferably, the shot place or the position and size of the subject within the screen are extracted and added to the accessory information together with the extracted information.
Preferably, the shot place or the position and size of the subject within the screen are extracted and added to the designated information together with the extracted information.
Preferably, a difference between image data of the actual shot image and image data simulating the shot image and which has been created from position information and camera information when the image was shot, and the map information is stored and when the image which is a result of searching is outputted, the stored difference is added to the image data simulating the shot image and which has been created from the position information, the camera information and the map information and the resulting image is output.
The present invention provides an image searching method, comprising the steps of: deriving information representing a distance from a screen recording an image; storing image data of the image as well as a deriving result allowed to be appendant to the image data of the image as accessory information; designating the information representing the distance from the image recording screen as reference data; determining a degree of agreement with the accessory information appendant to image data of the image; and reading an image having the degree of agreement not lower than a predetermined value.
Preferably, the specific information accompanying the image is information representing a distance from a screen recording the image, the accessory information is a result obtained by deriving the information representing the distance from the image recording screen, and the step of designating at least a portion of the accessory information as the searching condition comprises designating the information representing the distance from the image recording screen as reference data.
Preferably, if the image is a multi-stage focused image or a stereoscopic image, the information representing the distance from the image recording screen is derived from the multi-stage focused image or stereoscopic image.
Preferably, the information representing the distance from the image recording screen is derived from the position information when the image was shot and map information or from the position information, the map information and the camera information.
Preferably, an outline of the depth for each of the areas in the image recording screen is designated as reference data concerning the information representing the distance from the image recording screen.
Preferably, an outline of the depth is designated as reference data concerning the information representing the distance from the image recording screen by arranging patterns of subjects in a virtual map space.
Preferably, a difference between image data the actual recorded image and image data simulating the recorded image and which has been created from position information, camera information and map information that are relevant to a shooting time is stored and to output the image which is a result of searching, the stored difference is added to the image data simulating the recorded image and which has been created from the position information, camera information and map information that are relevant to the shooting time and the resulting image is output.
In order to attain the third aspect, the present invention provides an image processing method, comprising the steps of: extracting specific figures or specific subjects in a printed image during a previous printing; storing the extracted specific figures or subjects in a database as accessory information that is keyed to image processing conditions either alone or in combination with the image data; reading image data from a previous print or captured image data recording medium to extract the specific figures or specific subjects for processing prints upon reorder; determining a degree of agreement between the extracted specific figures or subjects and the accessory information stored in the database; searching for an image having the degree of agreement not lower than a predetermined value; and processing a hit image under the image processing conditions employed for the previous printing.
The present invention provides an image processing method, comprising the steps of: storing image processing conditions for a previous printing in a database in such a manner that they are keyed to image data; reading image data from a previous print or captured image data reading medium to extract a first specific figure or subject for processing prints upon reorder; also extracting a second specific figure or subject from the image data stored in the database; determining a degree of agreement between the first specific figure or subject and the second specific figure or subject; searching for an image having the degree of agreement not lower than a predetermined value; reading from the database the image data for a hit image together with the stored image processing conditions keyed to the image data; and subjecting the thus read image data to image processing under the image processing conditions as read from the data base.
The present invention provides an image processing method, comprising the steps of: extracting specific figures or specific subjects from a plurality of images within a single order; and performing image processing of image frames to yield prints that are finished to have quality within a predetermined range if the image frames are similar in terms of positions and sizes of the specific figures or subjects.